Stability of Feature Ranking Algorithms on Binary Data

Stability or robustness is a crucial yardstick for analyzing and evaluating feature selection algorithms which have become indispensible due to unprecedented advancements in knowledge data discovery and management. Stability of feature selection algorithms is taken as the insensitivity of the algorithm to perturbations in the training data with reference to the performance of the algorithm with all training data. In this work, we propose an algorithm for evaluating and quantifying the robustness of feature ranking algorithms and test three feature ranking algorithms: relief, diff-criterian and mutual information on four different real life binary data sets from text mining, handwriting recognition, medical diagnoses and medicinal sciences. We then analyze the stability profiles of feature selectors and determine how stability is a desirable characteristic of a feature ranking algorithm. We find that diff-criterian, and mutual information, outperform relief in stability.